The deformation such as translation, rotation and random scaling of local images in image recognition tasks is a complicated problem. An algorithm based on pre-training convolutional filters and Multi-Scale block Convolutional Neural Network (MS-CNN) was proposed to solve these problems. Firstly, the training dataset without labels was used to train a sparse autoencoder and get a collection of convolutional filters with characteristics in accord with the dataset and good initial values. To enhance the robustness and reduce the impact of the pooling layer for the feature extraction, a new Convolutional Neural Network (CNN) structure with multiple channels was proposed. The multi-scale block operation was applied to input image to form several channels, and each channel was convolved with corresponding size of filter. Then the convolutional layer, a local contrast normalization layer and a pooling layer were set to obtain invariability. The feature maps were put in the full connected layer and final features were exported for target recognition. The recognition rates of STL-10 database and remote sensing airplane images were both improved compared to traditional CNN. The experimental results show that the proposed method has robust performance when dealing with deformations such as translation, rotation and scaling.